Extract vm images #16 by pombredanne · Pull Request #20 · aboutcode-org/extractcode

pombredanne · 2021-04-06T22:07:54Z

This PR:

adds support to extract filesystems as files as found in VM images from VirtualBox, VMware and QEMU using libguestfs for Extract VM images #16
provides a new way to configure third-party binaries (using environment variables, plugins or PATH)

Update boilerplate

Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>

Signed-off-by: Steven Esser <sesser@nexb.com>

Fix .gitattributes

Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>

With this all supported archive formats with be tried. Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>

THis is a two step extraction using libguestfs to get a FS to a tarball which is then extractcode normally (hence dealing with links, device files and other permission oddities as a side effect). We support VDI (VirtualBox, VMDK (VMware) and QCOW2 (QEMU) Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>

Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>

Use virtualenv-embedded libraries

Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>

This will work even from a git archive or when git is not installed. Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>

Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>

This was they do not end up in the template CHANGELOG.rst Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>

Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>

We now load native libraries and executables from: 1. an envt. variable path 2. OR a locatin provider plugin 3. OR the PATH or we fail Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>

New configure

Reuse variables. Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>

Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>

We are not extracting symlinks, though it could be useful in the future for some cases. Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>

Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>

From aboutcode-org/typecode#20 Reported-by: Pierre Tardy <pierre.tardy@renault.com> Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>

Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>

Also support Python 3.9 Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>

Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>

And remove v prefix from fallback version Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>

Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>

Apply formatting and minor refactoring. Refine and carify documentation. Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>

pombredanne · 2021-06-01T10:28:06Z

All clear. Now merging for release.

rwmjones · 2021-06-01T20:42:22Z

A few comments about the patch in no particular order:

Instead of using guestfish it may or may not be worth considering using the API directly. We have a solid Python binding that covers the entire API. Example: https://libguestfs.org/guestfs-python.3.html#example-2:-inspect-a-virtual-machine-disk-image

For extracting things like partitions from disk images (ie. without processing the filesystem or individual files), we have the lighter weight tooling of libnbd / nbdkit (https://gitlab.com/nbdkit) / qemu-nbd. For example to copy out a partition from a raw disk image (to stdout), you might use:

nbdcopy -- [ nbdkit --filter=partition file disk.img partition=1 ] -

https://libguestfs.org/nbdkit-partition-filter.1.html#EXAMPLE
https://libguestfs.org/nbdcopy.1.html

To copy out a partition from a qcow2 file (again to stdout):

nbdcopy -- [ \
    nbdkit --filter=partition nbd \
        command=qemu-nbd arg=-f arg=qcow2 arg=/path/to/image.qcow2 \
        partition=1 ] -

https://libguestfs.org/nbdkit-nbd-plugin.1.html#Add-nbdkit-partition-filter-to-qemu-nbd
https://libguestfs.org/nbdcopy.1.html

pombredanne · 2021-06-01T21:40:05Z

@rwmjones thank you ++ for chiming in!

You wrote:

We have a solid Python binding that covers the entire API. Example: https://libguestfs.org/guestfs-python.3.html#example-2:-inspect-a-virtual-machine-disk-image

I considered it at first but then the uneven packaging and support for various Python versions across Linux distros made switch back to a CLI interface for now.

For extracting things like partitions from disk images (ie. without processing the filesystem or individual files), we have the lighter weight tooling of libnbd / nbdkit (https://gitlab.com/nbdkit) / qemu-nbd.

I looked at it in details too. The partitions are not enough and I would need to mound things afterwards otherwise.
Yet I really want the filesystems out so I can process a whole VM image analysis pipeline from a virtual disk in https://github.com/nexB/scancode.io/blob/main/scanpipe/pipelines/root_filesystems.py the same way we can already process whole Docker container images in https://github.com/nexB/scancode.io/blob/main/scanpipe/pipelines/docker.py

On a side note, I have been hit by https://bugs.launchpad.net/ubuntu/+source/libguestfs/+bug/1813662 and I posted there https://bugs.launchpad.net/ubuntu/+source/libguestfs/+bug/1813662/comments/25 ;)

steven-esser and others added 9 commits January 18, 2021 20:01

Merge pull request #19 from nexB/update-boilerplate

a2a1901

Update boilerplate

Default to 64 bits windows on CI

f46bc48

Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>

Use wheels embedded in virtualenv.pyz

182532f

Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>

Do not force an upgrade on virtualenv.pyz embeds

cd4e87b

Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>

Fix .gitattributes

51510cb

Signed-off-by: Steven Esser <sesser@nexb.com>

Merge pull request #21 from nexB/fix-gitattributes

386bb90

Fix .gitattributes

Use trace not debug for tracing

371c11e

Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>

Add new CLI option to support extracting all formats

056b6c1

With this all supported archive formats with be tried. Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>

pombredanne mentioned this pull request Apr 6, 2021

We should support having an input as an image aboutcode-org/scancode.io#132

Closed

pombredanne added 3 commits April 20, 2021 00:21

Remove empty READMEs from skeleton

4b519f8

Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>

Complete support for VM image extraction #16

1fbf4e3

Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>

Add CLI scripts copied from scancode-toolkit

aa5da29

Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>

pombredanne changed the title ~~[WIP] Extract vm images #16~~ Extract vm images #16 Apr 22, 2021

pombredanne added 16 commits April 23, 2021 07:36

Use correct paths in script

89d01bf

Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>

Expand supported VM images types

d0ba2c2

Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>

Merge pull request #24 from nexB/use-venv-embeds

1c84e9e

Use virtualenv-embedded libraries

Update markers syntax for pytest

d6fe59f

Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>

Add fallback version for setuptools_scm

ca6ab21

This will work even from a git archive or when git is not installed. Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>

Add note for setuptools_scam fallback version

1364bbb

Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>

Use azure-posix.yml for linux and macOS

be851b0

Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>

Adopt new configure script derived from ScanCode

4f0aecf

Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>

Add notes on customization

aa04429

Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>

Adopt new configure --dev convention

56ada8f

Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>

Clarify CHANGELOG to be Rst

0dbcdc9

Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>

Add skeleton release notes to README.rst

d21aef3

This was they do not end up in the template CHANGELOG.rst Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>

Improve README

ab707b5

Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>

Implement new envt. variables approach

8163e2a

We now load native libraries and executables from: 1. an envt. variable path 2. OR a locatin provider plugin 3. OR the PATH or we fail Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>

Merge pull request #25 from nexB/new-configure

a75737d

New configure

Streamline default kinds code

acb85c0

Reuse variables. Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>

pombredanne added 19 commits May 28, 2021 18:29

Format code

15d43d5

Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>

Work towards symlinks support

a63c849

We are not extracting symlinks, though it could be useful in the future for some cases. Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>

Format code and streamline license headers

a073286

Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>

Merge latest skeleton

ac9c505

Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>

Use new coomadn invocation for 7zip

6c00362

Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>

Improve kernel settings doc for VMs

f358194

Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>

Merge branch 'main'

93d03e0

Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>

Update format

3aeb2ec

Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>

Add Python 3.9 to Travis

2c412e8

Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>

Format and remove spurious spaces

69eec23

From aboutcode-org/typecode#20 Reported-by: Pierre Tardy <pierre.tardy@renault.com> Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>

Install and configure libguesfs in CI

1074c50

Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>

Update doc for release

08aa847

Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>

Skip vm image tests on Travis

28528df

Also support Python 3.9 Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>

Merge latest skeleton

bfde189

Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>

Bump to more modern version of setuptools_scm

0e09ad9

And remove v prefix from fallback version Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>

Add space for correct syntax

e339a70

Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>

Merge remote-tracking branch 'skeleton/main' into 16-vm-images

06bcb03

Sort imports

a0a1436

Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>

Improve failure reporting

9d9d1a1

Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>

pombredanne force-pushed the 16-vm-images branch from a7837d3 to 9d9d1a1 Compare May 31, 2021 20:03

pombredanne added 3 commits June 1, 2021 09:38

Add test documentation

6e765bf

Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>

Update documentation

bbbffbc

Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>

Improve documentation and readability

4a8ef69

Apply formatting and minor refactoring. Refine and carify documentation. Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>

pombredanne merged commit fabf1f2 into main Jun 1, 2021

pombredanne deleted the 16-vm-images branch June 1, 2021 10:28

Navigation Menu

Uh oh!

Uh oh!

Extract vm images #16 - #20

Extract vm images #16#20
pombredanne merged 50 commits into
mainfrom
16-vm-images

pombredanne commented Apr 6, 2021 •

edited

Loading

Uh oh!

pombredanne commented Jun 1, 2021

Uh oh!

rwmjones commented Jun 1, 2021

Uh oh!

pombredanne commented Jun 1, 2021

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Uh oh!

Uh oh!

Conversation

pombredanne commented Apr 6, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

pombredanne commented Jun 1, 2021

Uh oh!

rwmjones commented Jun 1, 2021

Uh oh!

pombredanne commented Jun 1, 2021

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

pombredanne commented Apr 6, 2021 •

edited

Loading